Design - Full Text Search

October 11, 2026

Last updated on October 27, 2026

 

The Search command can find text in paragraphs and in images. A search of many pages is slow, because OneMore must read each page from OneNote. This document explains how the text index makes the search fast. The index only narrows the search. The search gives the same results with or without it.

 

Related reading

Design - Page Identity explains the page key that this design uses. Design - Hashtags explains the scanner that shares its page reads with the index. TechNote - OneNote ID Stability explains why the index records the page ID.

 

1. Problem and scope

 

What is slow today

OneMore reads a page from OneNote through COM. One page read takes 35 to 90 ms. A search in all notebooks reads every page, one after the other. A user with 2,000 pages waits more than one minute for each search.

 

Goals

1. A search that uses the index gives the same results as a search that reads every page.
2. The index never writes to a page.
3. The user must turn the index on. Nothing is indexed before that.
4. Indexing does not make OneNote slow.
5. A search never fails because of the index. If the index cannot be used, the search reads the pages.

 

Non-goals

• Word stemming, ranking, or fuzzy matching.
• Text of a generated table of contents.
• Sharing the index between computers. Each computer has its own index.
• Searching text in attachments.

 

2. Concepts

 

 

Why trigrams

A trigram index finds a term anywhere in a paragraph, and ignores case. This is how the live search treats a term. A word index cannot find the term port in Support. The trigram index can.

 

The price is size. A trigram index is about three times the size of the text that it covers.

 

Why a stamp

OneNote can give a page a new page ID when a notebook is closed and opened again. The paragraph IDs then change too. The index records paragraph IDs, so it must index the page again. The stamp lets OneMore detect this.

 

3. Architecture

The index has a write side and a read side. The write side is a consumer of the background page pass. The read side is part of the Search dialog.

 

 

Architecture (Refresh)

 

 

Components

 

 

Why one shared page read

The hashtag scanner and the index both need the content of a page. The PageContentPass reads each page one time and gives it to both consumers. This halves the load on OneNote. The pass also keeps the throttle, so the index does not add its own.

 

4. Data model

 

The index lives in OneMore.db, in the same file as the identity catalog.

 

 

Data Model (Refresh)

 

 

 

Three triggers on index_text (insert, delete, update) keep index_fts current. Only IndexProvider writes index_text, because the triggers need FTS5.

 

5. Behavior

 

5.1 Background indexing

The background pass runs after the identity pass. For each page, the consumer applies these rules.

 

 

Background Indexing PlantUML (Refresh)

 

Notes on these rules:

• The modified time is compared for any difference, not only for a newer time. A page that the user restores from an older version is indexed again.
• A pass reads pages for at most 5 seconds. The consumer always reads at least one page, so the backlog always gets smaller.
• The consumer waits at least 25 ms after each page read.
• The first index of a large set of notebooks takes many passes. This is intended.
• The stamp holds the modified time that the page had when the pass began. This time can only be older than the page that was read. A page that the user edits during the pass is indexed again.

 

5.2 Build now

The user can start a build from Settings.

 

1. The build holds back the background consumer. The two do not run at the same time.
2. The build reads all chosen pages that are not current. It has no time budget.
3. The build reports progress after each page.
4. The user can cancel at any time.

 

Cancel is safe. Each page is one transaction. The background consumer continues from the place where the build stopped. A build only adds data. It never removes data.

 

5.3 Search

The Search dialog uses the index when these conditions are true:

 

• The user turned the index on.
• The scope is not a single page.
• Include table of contents is off.

 

 

Search Path PlantUML (Refresh)

 

The index finds a superset

BuildIndexQuery converts the query to an FTS5 expression. The expression finds every paragraph that can match, and maybe more. IndexedHits then tests each paragraph with the same regular expression as the live search. This is why the results are the same.

 

 

When the index cannot narrow the query, every paragraph is a candidate. The search is then correct, but it is not faster than reading the index table.

 

A page is answered by the index only when it is current

A page is current when its page ID and modified time equal the values in the OneNote hierarchy. Pages that are changed, or never indexed, are read from OneNote as before. The page that the user is on is always read, because it can have edits that OneNote did not record.

 

Text in images

The index always stores the text that OneNote recognized in images, with kind = 1. The query adds kind = 0 when Search text in images is off. The user can change the option without a new index.

 

5.4 Cleanup

 

 

The index keeps no text of a notebook that the user does not choose. That text would become old, and it would use space.

 

6. Loading FTS5

 

The System.Data.SQLite build in OneMore does not register FTS5. The file SQLite.Interop.dll contains FTS5 and exports sqlite3_fts5_init. IndexProvider loads it into each connection.

 

1. Enable extension loading on the connection.
2. Load the entry point from the interop file. Try the name first, then the architecture folder.
3. Disable extension loading again.

 

IndexProvider.IsAvailable runs this one time in a memory database and creates a trigram table. When this fails, the index is off and every search reads the pages.

 

A connection that did not load FTS5 can read all other tables. It fails on any statement that touches index_fts or the triggers. For this reason, only IndexProvider writes index_text.

 

7. Failure modes and open questions

 

Known limits

 

 

Decisions

 

Measurements

 

These numbers are an example from one database. They are not a guarantee.

 

 

The index_text table is larger than the text. The objectID column is the likely cause. It is not measured separately.

 

Here are measurements across nine test notebooks, again for demonstration purposes only. Your data storage will certainly vary depending on the size of your notebooks, pages, and image OCR results.

 

 

Where that 42 MB goes:

 

Sizes are the stored data per table in the snapshot. B-tree page overhead and secondary indexes account for the rest of the file.

 

 

The first six rows sum to about 36.5 MB. The remaining ~6 MB is an inference from the 42.7 MB file size, not a measured figure.

 

• The "index" is bigger than I reported. I said the text index was about 23.4 MB, but that was only the FTS5 trigram structure. The index_text table is also part of the feature and is another 13.3 MB. All search-related storage is therefore about 36 MB, or roughly 85% of the file.
• Hashtags, favorites, identity and pinned data are negligible.

 

Why index_text is 13.3 MB for only 7.4 MB of text

 

• The raw text is only about 7.4 MB (6.8 MB typed plus 0.6 MB OCR).
• The objectID column is the likely culprit. OneNote object IDs look like {1}, around 50 characters each. Over 105k paragraphs that comes to about 5 MB stored per paragraph row. I haven't measured the column's size directly.
• The remainder is small. The ordinal and kind integers and the IDX_index_text_pageKey index add little.

 

What this means

 

• Text is stored twice. Once in index_text and once, as trigrams, in the FTS index.
• Trigrams are expensive. Every 3-character window becomes an entry, which is why the index runs about 3x the text.
• The WAL is separate. It adds another ~7 MB on disk next to the main file.

 

Open questions

 

• Is it better to resolve the objectID when a hit is shown, and not store it for each paragraph? This could save about 5 MB in the database above.
• Does detail=column in FTS5 reduce the size and keep the results correct? Phrase queries could be affected.
• Does FTS5 load correctly on ARM64 and in the tray process? It is not verified.

 

8. Testing strategy

 

 

 

#omwiki #omdeveloper #omdesign

 

© 2026 Steven M Cohn. All rights reserved.

Please consider a sponsorship or one-time donation to support ongoing development